This work has the potential for wide-ranging applications in human-in-the-loop (e.g. We set the radius of agents to 0.3, the radius of The dataset will be made public. The only difference of our model's architecture to theirs is that we use agent-centric representations Then, we construct an edge from the agent that corresponds to the row to the "column agent" then compare this with the ground truth graph. The smaller the circle, the further it is into the future.

Optimizing the energy management within a smart grids scenario presents significant challenges, primarily due to the complexity of real-world systems and the intricate interactions among various components. Reinforcement Learning (RL) is gaining prominence as a solution for addressing the challenges of Optimal Power Flow (OPF) in smart grids. However, RL needs to iterate compulsively throughout a given environment to obtain the optimal policy. This means obtaining samples from a, most likely, costly simulator, which can lead to a sample e fficiency problem. In this work, we address this problem by substituting costly smart grid simulators with surrogate models built using Physics-Informed Neural Networks (PINN)s, optimizing the RL policy training process by arriving to convergent results in a fraction of the time employed by the original environment. Specifically, we tested the performance of our PINN surrogate against other state-of-the-art data-driven surrogates and found that the understanding of the underlying physical nature of the problem makes the PINN surrogate the only method that we studied capable of learning a good RL policy, in addition to not having to use samples from the real simulator. Our work shows that, by employing PINN surrogates, we can improve training speed by 50%, comparing to training the RL policy by not using any surrogate model, enabling us to achieve results with score on par with the original simulator more rapidly. Keywords: Smart Grids Control, Reinforcement Learning, Physics-informed Neural Networks, Active Network Management, Optimal Power Flow, Surrogate Models, Renewable EnergyRL Reinforcement Learning EA Expert agent PINN Physics-Informed Neural Networks ANN Artificial Neural Network OPF Optimal Power Flow ESS Energy Storage Systems SoC State of Change MAE Mean Absolute Error 1. Introduction Smart grids are a pivotal concept driving the current modernization of electrical networks, addressing the urgent need to reduce greenhouse gas emissions, enhance energy e fficiency, and improve grid stability through demand response mechanisms. The European Union aims to achieve 43% renewable energy generation by 2030 [1], and in 2021, the renewable energy share rose to 32 .1% [2]. Corresponding author Email address: julen.cestero@polimi.it Modern societies require advanced grids capable of predicting and mitigating the uncertainties associated with renewable energy sources.

We thank all reviewers for their careful reviews and many positive comments. We feel that the typos and minor issues are easily addressable and will be corrected. We will incorporate this analysis into a revision of the paper. We thank R1 for bringing this highly related work to our attention. That work focuses on environments for which mice have previously developed spatial maps.

This work has the potential for wide-ranging applications in human-in-the-loop (e.g. We set the radius of agents to 0.3, the radius of The dataset will be made public. The only difference of our model's architecture to theirs is that we use agent-centric representations Then, we construct an edge from the agent that corresponds to the row to the "column agent" then compare this with the ground truth graph. The smaller the circle, the further it is into the future.

This paper introduces an approach for developing surrogate environments in reinforcement learning (RL) using the Sparse Identification of Nonlinear Dynamics (SINDy) algorithm. We demonstrate the effectiveness of our approach through extensive experiments in OpenAI Gym environments, particularly Mountain Car and Lunar Lander. Our results show that SINDy-based surrogate models can accurately capture the underlying dynamics of these environments while reducing computational costs by 20-35%. With only 75 interactions for Mountain Car and 1000 for Lunar Lander, we achieve state-wise correlations exceeding 0.997, with mean squared errors as low as 3.11e-06 for Mountain Car velocity and 1.42e-06 for LunarLander position. RL agents trained in these surrogate environments require fewer total steps (65,075 vs. 100,000 for Mountain Car and 801,000 vs. 1,000,000 for Lunar Lander) while achieving comparable performance to those trained in the original environments, exhibiting similar convergence patterns and final performance metrics. This work contributes to the field of model-based RL by providing an efficient method for generating accurate, interpretable surrogate environments.

Q-learning is widely used to optimize wireless networks with unknown system dynamics. Recent advancements include ensemble multi-environment hybrid Q-learning algorithms, which utilize multiple Q-learning algorithms across structurally related but distinct Markovian environments and outperform existing Q-learning algorithms in terms of accuracy and complexity in large-scale wireless networks. We herein conduct a comprehensive coverage analysis to ensure optimal data coverage conditions for these algorithms. Initially, we establish upper bounds on the expectation and variance of different coverage coefficients. Leveraging these bounds, we present an algorithm for efficient initialization of these algorithms. We test our algorithm on two distinct real-world wireless networks. Numerical simulations show that our algorithm can achieve %50 less policy error and %40 less runtime complexity than state-of-the-art reinforcement learning algorithms. Furthermore, our algorithm exhibits robustness to changes in network settings and parameters. We also numerically validate our theoretical results.